Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTPBackendRef RequestMirror Conformance Test #2819

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

HTTPBackendRef RequestMirror Conformance Test #2819

wants to merge 4 commits into from
+105 −59

Conversation

dprotaso
Copy link
Contributor

@dprotaso dprotaso commented Feb 26, 2024 •
edited
Loading

What type of PR is this?

/kind cleanup
/kind test
/area conformance

What this PR does / why we need it

There's currently no conformance test for this feature which is 'Extended'

Additional notes

I dropped path logic in filter so we can re-use the same test for the backend

NONE

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 26, 2024
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Feb 26, 2024
@dprotaso
Copy link
Contributor Author

/kind cleanup
/kind test
/area conformance

@k8s-ci-robot k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/test area/conformance labels Feb 26, 2024
@dprotaso dprotaso changed the title HTTPBackendRef RequestMirror Conformance HTTPBackendRef RequestMirror Conformance Test Feb 26, 2024
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 27, 2024
@dprotaso
Copy link
Contributor Author

dprotaso commented Mar 7, 2024

adding approvers
/assign @arkodg @mlavacca @sunjayBhatia

adding reviewers
/cc @LiorLieberman @michaelbeaumont @Xunzhuo

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 1, 2024

Conformance reviewers/approvers - can I get review please @arkodg @mlavacca @sunjayBhatia @michaelbeaumont @LiorLieberman @Xunzhuo

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 3, 2024

I've added back all the test cases for the regular filter. The backend filter has a subset of these cases.

@arkodg
Copy link
Contributor

arkodg commented Apr 5, 2024 •
edited
Loading

has this been tested against any implementation ? afaik this isnt easily possible on envoy based implementations

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 5, 2024

has this been tested against any implementation ?

I ran these tests against Istio. The backendRef variant fails as expected because they don't support that feature.

@arkodg
Copy link
Contributor

arkodg commented Apr 5, 2024

imo lets add conformance tests if at least 1 implementation supports this case.

@robscott
Copy link
Member

robscott commented Apr 5, 2024

imo lets add conformance tests if at least 1 implementation supports this case.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 5, 2024 •
edited
Loading

imo lets add conformance tests if at least 1 implementation supports this case.

But then an implementation can't advertise the support this feature on the GatewayClass. They'll have to wait for a new gateway api release to do this.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

Then why is this functionality in the API? If we have the shape of feature in our API we can write conformance for it

@dprotaso
Copy link
Contributor Author

dprotaso commented Apr 5, 2024

Also the test is identical to the pre-split filter - so for this case I wouldn't be concerned with stability of the conformance test.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 17, 2024
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 18, 2024
@mlavacca
Copy link
Member

imo lets add conformance tests if at least 1 implementation supports this case.

But then an implementation can't advertise the support this feature on the GatewayClass. They'll have to wait for a new gateway api release to do this.

+1, I think it's pretty difficult to add conformance tests before at least one implementation supports a feature. If we were to do that, I think we'd need to introduce some concept of conformance test stability and consider untested conformance tests as a non-blocking reference point.

Then why is this functionality in the API? If we have the shape of feature in our API we can write conformance for it

I +1 what @arkodg and @robscott said about waiting to have at least one implementation on board with this feature. In my opinion, conformance tests are not intended to shape the API, but instead to validate API implementations. Regardless of the nature of the test (this test could be straightforward, as it shares the same logic of the requestMirror feature), we should wait for at least one implementation to properly run the tests, otherwise, the risk of having tests which are just a bunch of untested code is pretty high (I'm not saying it's the case of this specific PR, my concern is about the approach we want to have).

@shaneutt shaneutt requested a review from arkodg June 21, 2024 15:31
@mlavacca
Copy link
Member

mlavacca commented Aug 19, 2024 •
edited
Loading

In a backlog grooming session, we went through this issue and discussed the need to define the status of conformance tests (experimental/GA). Shane is going to take a look into it.

xref: #3009

/assign @shaneutt

shaneutt
shaneutt reviewed Aug 20, 2024
View reviewed changes
Copy link
Member

@shaneutt shaneutt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dprotaso Thanks for adding this!

In general, I think this looks good:

/approve

However, one thing that does give me some pause: We haven't really seen any support from other implementations saying that this test makes sense and works for them?

/hold

I can see at least two ways we can go about this. We could leave this PR on hold and do outreach, try to get some others to utilize it. However, we might need some kind of "test is experimental" flag so as to allow a soak period for new tests in lieu of immediate feedback from implementations? For the latter, we have a bit of a precedent in that we used an +experimental Go build tag for several months to stage in the version 2 of our conformance test suite (the one that introduced conformance reports). I'm not necessarily advocating for anything in particular yet, would like to know your thoughts @dprotaso ?

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 20, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dprotaso, shaneutt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 20, 2024
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 4, 2024
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shaneutt shaneutt shaneutt left review comments

@kflynn kflynn Awaiting requested review from kflynn

@LiorLieberman LiorLieberman Awaiting requested review from LiorLieberman

@michaelbeaumont michaelbeaumont Awaiting requested review from michaelbeaumont

@Xunzhuo Xunzhuo Awaiting requested review from Xunzhuo

@arkodg arkodg Awaiting requested review from arkodg

Assignees

@shaneutt shaneutt

@sunjayBhatia sunjayBhatia

@arkodg arkodg

@mlavacca mlavacca

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/conformance cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/test needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@dprotaso @arkodg @robscott @mlavacca @k8s-ci-robot @shaneutt @sunjayBhatia